Efficient spatial data partitioning for distributed $$k$$NN joins

Abstract Parallel processing of large spatial datasets over distributed systems has become a core part modern data analytic like Apache Hadoop and Spark. The general-purpose design these does not natively account for the data’s attributes results in poor scalability, accuracy, or prolonged runtimes. Spatial extensions remedy problem introduce recognition operations. At extension, locality-preserving partitioner determines how to spatially group dataset’s objects into smaller chunks using system’s available resources. Existing rely on sampling often mismanage non-spatial by either overlooking their memory requirements excluding them entirely. This work discusses various challenges that face partitioning proposes novel effectively queries datasets. For evaluation, proposed is integrated with well-known k -Nearest Neighbor ( $$k$$ k NN) join query. Several experiments evaluate proposal real-world Our approach differs from existing proposals (1) accounting unique traits without sampling, (2) considering computational overhead required handle data, (3) minimizing partition shuffles, (4) computing optimal utilization resources, (5) achieving accurate results. contributes through providing comprehensive discussion problems facing processing, development technique in-memory an effective, built-in, load-balancing methodology reduces query skews, Spark-based implementation NN Experimental tests show up $$1.48$$ 1.48 times improvement runtime as well accuracy